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Abstract. It is shown that membership in rational subsets of wreath products 
H I V with H a finite group and V a virtually free group is decidable. On the 
other hand, it is shown that there exists a fixed finitely generated submonoid in 
the wreath product Z ! Z with an undecidable membership problem. 



1 Introduction 

The study of algorithmic problems in group theory has a long tradition. Dehn, in his 
seminal paper from 1911 |6|, introduced the word problem (Does a given word over 
the generators represent the identity?), the conjugacy problem (Are two given group 
elements conjugate?) and the isomorphism problem (Are two given finitely presented 
groups isomorphic?), see [24 1 for general references in combinatorial group theory. 
Starting with the work of Novikov and Boone from the 1950's, all three problems were 
shown to be undecidable for finitely presented groups in general. A generalization of 
the word problem is the subgroup membership problem (also known as the generalized 
word problem) for finitely generated groups: Given group elements g, gi, . . . , g„, does 
g belong to the subgroup generated by gi , . . . , gn ? Explicitly, this problem was intro- 
duced by Mihailova in 1958, although Nielsen had akeady presented an algorithm for 
the subgroup membership problem for free groups in his paper from 1921 1271 . 

Motivated partly by automata theory, the subgroup membership problem was fur- 
ther generalized to the rational subset membership problem. Assume that the group G 
is finitely generated by the set X (where a G X if and only if a^^ £ X). A finite 
automaton A with transitions labeled by elements of X defines a subset L{A) C G in 
the natural way; such subsets are the rational subsets of G. The rational subset mem- 
bership problem asks whether a given group element belongs to L{A) for a given finite 
automaton (in fact, this problem makes sense for any finitely generated monoid). The 
notion of a rational subset in a monoid can be traced back to the work of Eilenberg and 
Schiitzenberger from 1969 [9|. Other early references are [111121 . Rational subsets of 
groups also found applications for the solution of word equations (here, quite often the 
term rational constraint is used) [7,20 1. In automata theory, rational subsets are tightly 
related to valence automata: For any group G, the emptiness problem for valence au- 
tomata over G (which are also known as G-automata) is decidable if and only if G has 
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a decidable rational subset membership problem. See II10I17I18I for details on valence 
automata and G-automata. 

For free groups, Benois (2] proved that the rational subset membership problem is 
decidable using a classical automaton saturation procedure (which yields a polynomial 
time algorithm). For commutative groups, the rational subset membership can be solved 
using integer programming. Further (un)decidability results on the rational subset mem- 
bership problem can be found in [21 1 for right-angled Artin groups, in [28 1 for nilpotent 
groups, and in ||23l for metabelian groups. In general, groups with a decidable rational 
subset membership problem seem to be rare. In [22] it was shown that if the group G 
has at least two ends, then the rational subset membership problem for G is decidable 
if and only if the submonoid membership problem for G (Does a given element of G 
belong to a given finitely generated submonoid of G?) is decidable. 

In this paper, we investigate the rational subset membership problem for wreath 
products. The wreath product is a fundamental operation in group theory. To define the 
wreath product H I G of two groups G and H, one first takes the direct sum K = 
^g^Q H of copies of H, one for each element of G. An element g E G acts on K by 
permuting the G-copies of H according to the left action of g on G. The corresponding 
semidirect product is the wreath product H I G. 

In contrast to the word problem, decidability of the rational subset membership 
problem is not preserved under wreath products. For instance, in ll23l it was shown that 
for every nontrivial group H, the rational subset membership problem for H I {Z x Z) 
is undecidable. The proof uses an encoding of a tiling problem, which uses the grid 
structure of the Cayley graph of Z x Z. 

In this paper, we prove the following two new results concerning the rational subset 
membership problem and the submonoid membership problem for wreath products: 

(i) The submonoid membership problem is undecidable for Z I Z. The wreath product 
Z ; Z is one of the simplest examples of a finitely generated group that is not finitely 
presented, see II5I4I for further results showing the importance of Z ? Z. 

(ii) For every finite group H and every virtually free group0 V, the group H I V has 
a decidable rational subset membership problem; this includes for instance the fa- 
mous lamplighter group Z2 I Z. 

For the proof of (i) we encode the acceptance problem for a 2-counter machine (Minsky 
machine ||25l ) into the submonoid membership problem for Z I Z. One should remark 
that Z ; Z is a finitely generated metabelian group and hence has a decidable subgroup 
membership problem [29 30|. For the proof of (ii), an automaton saturation procedure 
is used. The termination of the process is guaranteed by a well-quasi-order (wqo) which 
refines the classical subsequence wqo considered by Higman [15 1. 

Wqo theory has also been applied successfully for the verification of infinite state 
systems. This research led to the notion of well-structured transition systems ifTTl . An 
application in formal language theory is the decidability of the membership problem 
for leftist grammars [26|. Usually, a disadvantage of using wqo theory is that it does 
not yield algorithms with good complexity bounds. In the context of well-structured 
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transition systems, several natural reachability problems (e.g. for lossy channel sys- 
tems) were shown to be not primitive recursive 1131321 . Also the membership problem 
for leftist grammars was shown be not primitive recursive (16]. The complexity status 
for the rational subset membership problem for wreath products H I V {H finite, V 
virtually free) remains open. Actually, we do not even know whether the rational subset 
membership problem for the lamplighter group Z2 i Z is primitive recursive. 

As mentioned earlier, the rational subset membership problem is undecidable for 
every wreath product Hl{ZxZ), where H isa nontrivial group. We conjecture that this 
can be generalized to the following result: For every nontrivial group H and every non- 
virtually free group G, the rational subset membership problem for HlG is undecidable. 
The reason is that the undecidability proof for H l{Z x Z) ll23l only uses the grid-like 
structure of the Cayley graph of Z x Z. In (19] it was shown that the Cayley graph of a 
group G has bounded tree width if and only if the group is virtually free. Hence, if G is 
not virtually free, then the Cayley-graph of G has unbounded tree width, which means 
that finite grids of arbitrary size appear as minors in the Cayley-graph of G. One might 
therefore hope to reduce again a tiling problem into the rational subset membership 
problem for H I G (for H non-trivial and G not virtually free). 

Our decidability result for the rational subset membership problem for wreath prod- 
ucts H I V with H finite and V virtually free can be also interpreted in terms of tree 
automata with additional data values. Consider a tree walking automaton operating on 
infinite rooted trees. Every tree node contains an additional data value from a finite 
group such that all but finitely many nodes contain the group identity. Besides navigat- 
ing in the tree, the tree automaton can multiply (on the right) the group element from 
the current tree node with another group element (specified by the transition). The au- 
tomaton cannot read the group element from the current node. Our decidability result 
basically says that reachability for this automaton model is decidable. 

2 Rational subsets of groups 

Let G be a finitely generated group and X a finite symmetric generating set for G 
(symmetric means that X is closed under taking inverses). For a subset B C G we 
denote with B* the submonoid of G generated by B. The subgroup generated by B is 
{B). The set of rational subsets of G is the smallest set that (i) contains all finite subsets 
of G and (ii) that is closed under union, product, and *. Alternatively, rational subsets 
can be represented by finite automata. Let A — {Q,G,E,qo,Qp)hea finite automaton, 
where transitions are labeled with elements of G: Q is the finite set of states, £ Q is 
the initial state, Qp C Q is the set of final states, and i^CQxGxQisa finite set 
of transitions. Every transition label g E G can be represented by a finite word over the 
generating set X. In this way, A becomes a finite object. The subset L{A) C G accepted 
by A consists of all group elements (713253 • • • ffn such that there exists a sequence of 
transitions (go, 51j (91, ff2, 92), (92, 53, 93), ■ • ■ , (gn-i, g™, gn) e E v/ith q„ £ Qf- 
The rational subset membership problem for G is the following decision problem: 

INPUT: A finite automaton A as above and an element g £ G. 
QUESTION: Does g £ L{A) hold? 
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Since g £ L{A) if and only if Iq G L{A)g^^, and L{A)g^^ is rational, too, the rational 
subset membership problem for G is equivalent to the question of deciding whether a 
given automaton accepts the group identity. 

The submonoid membership problem for G is the following decision problem: 

INPUT: Elements g, gi, . . . , g„ e G. 
QUESTION: Does 5 G {51, . . . , g„}* hold? 

Clearly, decidability of the rational subset membership problem for G implies decid- 
ability of the submonoid membership problem for G. Moreover, the latter generalizes 
the classical subgroup membership problem for G (also known as the generalized word 
problem), where the input is the same as for the submonoid membership problem for G 
but it is asked whether g & {gi, ■ ■ ■ , gn) holds. 

In our undecidability results in Section|5] we will actually consider the non-uniform 
variant of the submonoid membership problem, where the submonoid is fixed, i.e., not 
part of the input. 

3 Wreath products 

Let G and H be groups. Consider the direct sum 

where Hg is a copy of H. We view K as the set H^'^'> {/ g iJ"^ | f^^{H \ 
{l/f }) is finite} of all mappings from G to H with finite support together with pointwise 
multiplication as the group operation. The group G has a natural left action on H^^^ 
given by 

where / G H^'^^ and g,a £ G. The corresponding semidirect product H^*^^ x G is the 
wreath product H iG.ln other words: 

- Elements of H } G are pairs where 5 G G and / e H'-'^\ 

- The multiplication in H I G is defined as follows: Let (/i,5i), (/2,32) G H iG. 
Then (/i, gi)(/2, 32) = if, 9192), where /(a) = /i(a)/2(gr^a). 

The following intuition might be helpful: An element (/, g) E H I G can be thought 
of as a finite multiset of elements of H \ {1h} that are sitting at certain elements of G 
(the mapping /) together with the distinguished element g E G, which can be thought 
of as a cursor moving in G. If we want to compute the product (/i, gi)(/2, .92), we do 
this as follows: First, we shift the finite collection of i/-elements that corresponds to 
the mapping /2 by gi: If the element h G H\{1h} is sitting at a G G (i.e., f2{a) = h), 
then we remove h from a and put it to the new location gia G H. This new collection 
corresponds to the mapping : a M> /2((7f ^a). After this shift, we multiply the two 
collections of iJ-elements pointwise: If in a G G the elements hi and /12 are sitting 
(i.e., /i(a) — hi and /2(a) = /12), then we put the product /11/12 into the location a. 
Finally, the new distinguished G-element (the new cursor position) becomes gig2- 
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Proposition 1. Let K be a subgroup of G of finite index m and let H be a group. Then 
} K is isomorphic to a subgroup of index m in H I G. 

Proof. Let T be a set of right coset representatives for G/K; it has m elements. The 
action of G on H'^^^ restricts to an action of K on H'-^^ and so if ^"^^ x if is a subgroup 
of HlG. There is a i^T-equivarianjl group isomorphism a: H^'^^ — >■ (iJ^)'^) given by 
[a{f){k)]{t) = f{kt), where / e H^^\ k & K,wdt e T. This a is indeed bijective; 
the inverse a'^ is given by [a-^{f)]{kt) = [f(k)]{t) for / e {H'^)^^\ k e K, and 
t G T (which has finite support because T is finite and / has finite support). That a is 
A'-equivariant follows from 

[ka{f)ik'm = [a{f){k-'k'm - /(fc-ifc't) = [kf]{k't) = [a{kf){k'm. 

It follows that H"'lK9:^ {H^Y^^ >o K ^ H'^^^ x K. 

It thus remains to prove that ii^*^^ xi K has index m in H I G. Indeed, let e e ii^^^ 
be the map sending all of G to the identity of H. Then the elements of the form (e, t) 
with t GT form a set of right coset representatives of H^'^'> x K in H iG. Indeed, it is 
easy to see that these elements are in distinct cosets. If g — kt with k G K and t € T, 
then (/, g) = (/, fc)(e, t), which is in the coset of (e, t). □ 

4 Decidability 

We show that the rational subset membership problem is decidable for groups G = 
H I V, where H is finite and V is virtually free. First, we will show that the rational 
subset membership problem for G = H IF2, where F2 is the free group generated by a 
and h, is decidable. For this we make use of a particular well-quasi-order 

4.1 A well-quasi-order 

Recall that a well-quasi-order on a set ^ is a reflexive and transitive relation < such 
that for every infinite sequence ai, 02, as, . . . with ai G A there exist i < j such that 
ai :< aj. In this paper, ^ will be always antisymmetric as well; so ^ wiU be a well 
partial order 

For a finite alphabet X and two words u,v G X*, we write u ^ v if there exist 
fo, • • • , Vn e X*, ui, . . . , M„ G X such that v = vqUiVi ■ ■ ■ w„w„ and u — ui • ■ • u„. 
The following theorem was shown by Higman ifTSl (and independently Haines lfT4l ). 

Theorem 1 (Higman's Lemma). 77ie order ^ on X* is a well-quasi-order 

Let G be a group. For a monoid morphism a : X* G and u,v G X* let u :<a v if 
there is a factorization v — vqUiVi ■ ■ ■ m„w„ with vq, . . . ,Vn G X*, wi, . . . , w„ G X, 
u — ui ■ ■ ■ Un, and a{vi) — 1 for < i < n. It is easy to see that ^q, is indeed a 
partial order on X*. Furthermore, let be the partial order on X* with u :<g v if 
V = VqUiVi ■ ■ ■ UnVn for some Vq, . . . ,Vn G X*, Ml, . . . , u„ G X, and u = ui • • • w„ 

' A i('-equivariant group isomorphism a : H^'^^ — > [H'^Y^^ is an isomorphism that commutes 
with the action of K: ka{f) — a(kf). 
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such that a{vi) = 1 for every morphism a : X* — !> G and < i < n. Note that if G is 
finite, there are only finitely many morphisms a : X* — G. This means that for given 
X and G, we can construct a finite automaton for the upward closure U C X* of {e} 
with respect to Since for w — wi ■ ■ ■ Wn, wi, . . . ,Wn £ X, the upward closure 
of w equals Uwi ■ ■ ■ UwnU, we can also construct a finite automaton for the upward 
closure of any given singleton provided that G is finite. In the latter case, we can also 
show that is a well-quasi-order: 

Lemma 1. Let G be a group. Then the following statements are equivalent: 

(i) {X* , :<q) is a well-quasi-order for each finite alphabet X. 

( ii) For every n G N, there is a k Cz N with \ {gi , . . . , 5„) | < kfor all gi, . . . , g„ G G. 

Proof. Suppose [(ii)| does not hold. Then there is a finite alphabet X and a sequence of 
morphisms ai, a2, ••■ ^ X* G such that \ai{X*)\ > i for each i > 1. We inductively 
define a sequence of words wi,W2, . ■ . G X*. Choose wi = e and suppose wi, . . . ,Wi 
have been defined. Since |ai4.i(X*)| > i + 1, we can choose Wi+i G X* to be a 
word such that ai+i{wi+i) is outside of {ai+i{wi) , . . . , ai+i{wi)}. We claim that the 
words wi,W2, . ■ . are pairwise incomparable with respect to :<g- Observe that u :<g v 
implies a{u) = a{v) for any morphism a : X* G. Since for any i, j G N, i < j, the 
construction guarantees aj{wj) ^ aj{wi), the words are pairwise incomparable. 

Suppose |(ii)| does hold and let X he a finite alphabet. First, we claim that there 
is a finite group H such that coincides with ^^f. By |(ii)| there are only finitely 
many non-isomorphic groups that appear as a{X*) for morphisms a: X* G, say 
Hi, ... , Hm, and each of them is finite. For H = Hi x • ■ • x iJ,„, we have 

ker(Q:) — ker(a). 

Hence, <g coincides with <h- There are only finitely many morphisms a : X* H, 
say Q!i, . . . , a£. If /3 : X* H^ is the morphism with P{w) = {ai{w), . . . , ai{w)), 
then 

Pi ker(a) = ker(/3). 

a-.X'-^H 

Thus, <H coincides with <p. Therefore, it suffices to show that <fi is a well-quasi- 
order 

Let wi,W2t . ■ . G X* be an infinite sequence of words. Since H^ is finite, we can 
assume that all the Wi have the same image under /3; otherwise, choose an infinite 
subsequence on which /? is constant. Consider the alphabet Y = X x H^. For every 
w G X*, w = oi • ■ • Or, let w G Y* be the word 

w = {ai, I3{ai)){a2, 13{aia2)) ■ ■ ■ {ar,l3{ai ■ • -a^)). (1) 

Applying Higman's Lemma to the sequence iI;i,W2, . ■ . yields indices i < j such that 
Wi :< Wj. This means Wi = u[ ■ ■ ■ u'^, Wj = v'f^u'iv[ ■ ■ ■ u'^v'^. for some u'j^, . . . , lij, G 
v'q, . . . ,v'j. G Y* . By definition of Wi and Wj, we have u'^ = (ws, hs) for 1 < s < r, 
where hs = /3(wi • • • Ug) and Wi = ui ■ ■ ■ Ur- Let tti : F* — > X* be the morphism 
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extending the projection onto the first component, and let Vg = t^i{v's) for < s < r. 
Then clearly wj — vqUiVi ■ ■ ■ UrVr- We claim that = 1 for < s < r, from 

which Wi Wj and hence the lemma follows. Since Wj is also obtained according to 
i^, we have 

I3{ui ■ ■ ■ Us+l) = hs+l = I3{vqUiVi ■ ■ ■ UsVsUs+l) 

for 0<s<r — 1. By induction on s, this allows us to deduce f3{vs) = 1 for < s < 
r — 1. Finally, f3{wi) = I3{wj) entails 

implying /3(ur) = 1- □ 



4.2 Loops 

Let G = H I F2 and fix free generators a,b e i^2- Recall that elements of G are pairs 
(fc, /), where k ^ K = ^g^p^ H and / G F2. In the following, we simply write kf for 
the pair {k, f). Fix an automaton A — {Q, G,E,qQ,QF) with labels from G for the rest 
of Section |4] We want to check whether 1 e L{A). Since G is generated as a monoid 
by -ff U {a, a~^, b, b~^}, we can assume that E C Q x {H (J {a, a^^,b, b^^}) x Q. 

A configuration is an element of Q x G. For configurations (p, 51), (5,52), we 
write (p, gi) — >yi {q, §2) if there is a (p, g,q) ^ E such that 32 = 515- For elements 
f,9 G F2, we write / < g (/ < g) if the reduced word representing / is a (proper) 
prefix of the reduced word representing g. We say that an element / e F2 \ {1} is 0/ 
type X G {a, a^^, b, b^^} if the reduced word representing / ends with a;. Furthermore, 
1 G i^2 is of type 1. Hence, the set of types isT = {1, a, a~^, b, b~^}. When regarding 
the Cayley graph of F2 as a tree with root 1, the children of a node of type t are of 
the types G{t) — {a, a^^, 6, 6^^} \ {t^^}- Clearly, two nodes have the same type if 
and only if their induced subtrees of the Cayley graph are isomorphic. The elements of 
D — {a, a^^, b, b^^} will also be called directions. 

Let p,q ^ Q and t G T. A sequence of configurations 

(91, fcl/l) -^A (92, ^2/2) ->-A >A (qn.knfn) (2) 

is called a well-nested {p, q)-computation in t if 

(i) qi^p and g„ = g, 

(ii) /i = fn is of type t, and 

(iii) fi > fi for 1 <i <n. 

We define the effect of the computation to be fi^k^^knfi G K. Hence, the effect 
describes the change imposed by applying the corresponding sequence of transitions, 
independently of the configuration in which it starts. For / G -F2, let |/| be the length 
of the reduced word representing /. The depth of the computation is the maximum 
value of |/rV»l for 1 < i < n. Of course, if fi — /„ — 1, condition |(iii)| is satisfied 
automatically. Hence, we have 1 G L{A) if and only if for some q E Qp, there is a 
well-nested (go, g)-computation in 1 with effect 1. 
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For d E C{t), a well-nested {p, g) -computation (|2]l in i is called a {p, d, q)-loop in t 
if in addition fid < fi for 1 < i < n. Note that there is a (p, d, g)-loop in t that starts 
in (p, kf) (where / has type t) with effect e and depth m if and only if there exists a 
(p, d, q)-loop in t with effect e and depth m that starts in (p, t). 

Given p^q E Q, t G T, d G C(t), it is decidable whether there is a {p,d,q)- 
loop in t: This amounts to checking whether a given automaton with input alphabet 
{a, ,b,b~^} accepts a word representing the identity of F2 such that no proper 
prefix represents the identity of F2. Since this can be accomplished using pushdown 
automata, we can compute the set 

Xt — {{p, d,q) e Q X C{t) X Q I there is a (p, d, q)-\oo^ in t}. 
4.3 Loop patterns 

Given a word w = (pi, di, qi) ■ ■ ■ {pn, dn, qn) G X^, a loop assignment for is a 
choice of a (p^, d^, qi)-loop in t for each position i, \ < i < n. The effect of a loop 
assignment is ei • • • e„ G K, where G -fC is the effect of the loop assigned to position 
i. The depth of a loop assignment is the maximum depth of an appearing loop. A loop 
pattern for t is a word w G X* that has a loop assignment with effect 1. The depth 
of the loop pattern is the minimum depth of a loop assignment with effect 1. Note that 
applying the loops for the symbols in a loop pattern (pi, di, qi) ■ • • (p„, (i„, q„) does 
not have to be a computation: We do not require qi = Pi+i- Instead, the loop patterns 
describe the possible ways in which a well-nested computation can enter (and leave) 
subtrees of the Cayley graph of F2 in order to have effect 1. The sets 

Pt = {w E XI I ui is a loop pattern for t} 

for t eT will therefore play a central role in the decision procedure. 
Recall the definition of the well-quasi-order from Section |4T| 

Lemma 2. For each t E T, the set Ft is an upward closed subset of with respect to 

Proof. Since i^T is a direct sum of copies of H, the orders <h and coincide. It 
therefore suffices to show that Ft is upward closed with respect to <k- Let u E Ft and 
w diK V, V E Xf, meaning v = vqUiVi ■ ■ ■ u„w„ with u = wi • • • u„ and a{vi) = 1, 
< i < n, for every morphism a : X^ — >■ K. Since u E Ft, there is a loop assignment 
for each Ui, 1 < i < n, with effect Ci such that ei • • ■ e„ = 1. By construction of Xt, 
for each {p, d, q) E Xt, there is a {p, d, g)-loop, say ip.d,q, in t. Let ip : Xt Khe the 
morphism such that for each {p, d, q) E Xt, f{{p, d, q)) is the effect of £p,d.q- Choosing 
ip.d,q for each occurrence of {p, d, q) in a subword Vi and reusing the loop assignments 
for the Ui defines a loop assignment for v. Since (p{vi) = 1 for < i < n, the effect of 
this loop assignment is (f{vo)ei(f{vi) ■ ■ ■ en^{vn) — ei ■ ■ ■ e„ — 1. Hence, v E Ft. □ 

Since is a well-quasi-order, the previous lemma already implies that each Ft is a 
regular language. On the one hand, this follows from the fact that the upward closure of 
each singleton is regular. On the other hand, this can be deduced by observing that :<h 
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is a monotone order in the sense of |i8J. Therein, Ehrenfeucht, Haussler, and Rozenberg 
show that languages that are upward closed with respect to monotone well-quasi-orders 
are regular. Our next step is a characterization of the sets Pt that allows us to compute 
finite automata for them. In order to state this characterization, we need the following 
definitions. 

Let X, Y be alphabets. A regular substitution is a map a: X ^ 2^ such that a{x) 
is a regular language for every x E X. For w £ X*, w = wi ■ ■ ■ Wn, Wi G X, let 
a{w) = Ri ■ ■ ■ Rn, where cr(wi) = Ri for 1 < i < n. Given a set i? C Y* and a 
regular substitution o- : X -> let (J^^{R) = {w <E X* \ a{w) n i? ^ 0}. Note that 
if R is regular, then (j^^{R) is regular as well [31^, Proposition 2.16], and an automaton 
for (j^^{R) can be constructed effectively from an automaton for R and automata for 
the a{x)^ 

The alphabet Yt is given by 

Yt=XtU{{QxHxQ)r\E). 
The morphism tt^ : Y^ X^ is the projection onto X^, meaning 



T^t{y) 



y for 2/ e Xt 

e for yeYt\Xt. 



The morphism h't : Y^* H is defined by 

i^tiiP, d, q)) = 1 for (p, d, q) S Xt 
vt{{p, h, q)) = h for {p, h, q) eYt\ Xf 

For p,q E Q and t E T, define the regular set 

K.q = {{P0,9l,Pl){Pl,92,P2) ■ ■ ■ {Pn~l,9n,Pn) £ Yf* \ PQ = p, _p„ ^ q} . 

Given t E T and d E C{t), the regular substitution Cf ^ : Xt 2^^ is defined by 

aM((p,d,'7))=Ui^P'.9' I {P,d,p'),{q\d-\q)EE} 
at,d{{p, u, q)) = {e} for u E C{t) \ {d}. 

Given two tuples, {Ut)teT and (Vt)tgT with Ut,Vt C X^, we write {Ut)t^T < {Vt)t&T 
if Ut C Vt for each t E T. 



Lemma 3. {Pt)t^T is the smallest tuple such that for every t E T we have e E Pt and 

n K'(^d)n^.,-i(i)) CP,. (3) 

deC(t) 

* In f311, it is shown that the class of regular languages is closed under arbitrary inverse substi- 
tuitions. Moreover, the construction is effective if cr is a regular substitution. 
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Proof. For each i e N, let ' C be the set of loop patterns for t whose depth is at 
most i. Then pj;^^ — {e} and 

dec(t) 

The lemma follows since Pt = Ui>o -f'i*'*- '-' 

Given a language L C X^*, let itt = G -'^t* I " ^ff « for some u e L}. 

Theorem 2. 77ie rational subset membership problem is decidable for every group G = 
H } F, where H is finite and F is a finitely generated free group. 

Proof. Since H I F is a subgroup of H IF2 (since F is a subgroup of F2), it suffices to 
show decidability for G = H I F2. First, we compute finite automata for the languages 
Pt. We do this by initializing C//*'^ := {e}tt for each t £ T and then successively 
extending the sets , which are represented by finite automata, until they equal Pt: 
If there is a i S T and a word 

we set C//'+^^ [/f ^ U {u;}tt and C/i'+^^ C/i'^ for u G T\ {t}. Otherwise we stop. 
By induction on i, it follows from Lemma|2]and Lemma[3]that C Pf. 

(i+l) (i) (i) 

In each step, we obtain by adding new words to Uf . Since the sets 

are upward closed by construction and there is no infinite (strictly) ascending chain 
of upward closed sets in a wqo, the algorithm above has to terminate with some tuple 
(C/j'^'')fgT- This, however, means that for every t e T 

Since on the other hand e e U^^^ and U^^^ C Pt, LemmaSyields C//''^ = Pt. 

Now we have 1 G L{A) if and only if 7rj'^(Pi) n I'f ^(1) n ^ ^ for some 
q e Qf, which can again be checked by constructing and analyzing finite automata. 

□ 

Theorem 3. The rational subset membership problem is decidable for every group H I 
V with H finite and V virtually free. 

Proof. This is immediate from Theorem |2] and Proposition [T] because if F is a free 
subgroup of index m in V, then iJ }F is isomorphic to a subgroup of index m in HlV 
and decidability of rational subset membership is preserved by finite extensions 0131181 . 

□ 
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5 Undecidability 



In this section, we will prove the second main result of this paper: The wreath product 
IXL contains a fixed submonoid with an undecidable membership problem. Our proof is 
based on the halting problem for 2-counter machines (also known as Minsky machines), 
which is a classical undecidable problem. 

5.1 2-counter machines 

A 2-counter machine (also known as Minsky machine) is a tuple C — (Q, go, qf,S), 
where 

- Q is a finite set of states, 

- go G Q is the initial state, 

- Qf E Q is the final state, and 

- (5 C (Q \ {(//}) X {cq, ci} X {+1, —1, = 0} X Q is the set of transitions. 

The set of configurations is Q x N x N. On this set we define a binary relation -^c 
as follows: {p, mo, nii) {q, no, rii) if and only if one of the following three cases 
holds: 

- There exist i G {0, 1} and a transition {p, Ci, +1, q) <E 6 such that rii — rrii + 1 and 
m^i = mi_i. 

- There exist i G {0, 1} and a transition [p, Ci,—l,q) G 5 such that rii — rrii ~ 1 (in 
particular, we must have > 0) and = mi^i. 

- There exist i G {0, 1} and a transition [p, a, = 0, g) G (5 such that rn — rm — 

and ni-i = toi-;. 

It is well known that every Turing-machine can be simulated by a 2-counter machine, 
see e.g. ||251 . In particular, we have: 

Theorem 4. There exists a fixed 2-counter machine C = {Q, go, g/, 5) such that the 
following problem is undecidable: 

INPUT: Numbers m, n G N. 

QUESTION: Does (go, m, n) -J-^ (g/, 0, 0) hold? 

5.2 Submonoids of Z ? Z 

In this section, we will only consider wreath products of the form H I Z. An element 
{f,m) G HlZi such that the support of / is contained in the interval [a, b] (with a,b G Z) 
and 0, m G [a, b] will also be written as a list [/(a), . . . , /(6)], where in addition the 
element /(O) is labeled by an incoming (downward) arrow and the element /(m) is 
labeled by an outgoing (upward) arrow. 

In this section, we will construct a fixed finitely generated submonoid of the wreath 
product Z I Z with an undecidable membership problem. 
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Let C — {Q, Qo, qf,S) be the 2-counter machine from Theorem^] Without loss of 
generality we can assume that there exists a partition Q = Qo U Qi such that go G Qo 
and 

<5 C (Qo X {co} X {+1, -1, = 0} X Qi) U (Qi x {ci} x {+1, -1, = 0} x Qo)- 

In other words, C alternates between the two counters. Hence, a transition {q, Ci, x,p) 
can be just written as {q,x,p). Let 

S^QU{c,#}. 

Let be the free abelian group generated by U. First, we will prove that there is a 
fixed finitely generated submonoid M of the wreath product Z^lZ with an undecidable 
membership problem. Let 

An infinite presentation for Z^ I Z is 

Z-^ ; Z (Z", a I [a"sa-", a"ta"™] = 1 (n, m e Z, s, t e S)). 

Here, a generates the right Z-factor of the wreath product Z"^ I Z. In the following, 
we will freely switch between the description of elements of Z"^ I Z by words over 
{S U {a})^^ and by pairs from K yi 1,. For a finite-support mapping f ^ K, m & "L, 
and cc G 17, we also write /(m, x) for the integer f{m){x). 

Our finitely generated submonoid M of Z^ ? Z is generated by the following ele- 
ments. The right column shows the generators in list notation, where elements of the 
free abelian group Z^ are written additively, i.e., as Z-linear combinations of elements 
of Z": 



p"^a#a^#a(7 for (p, = 0, £ (5 
p^^a^aca^qa^^ for {p, +1, q) E S 
p-'^a#a^qa^c-^a'^ for {p, -l,q) e 5 



qj'a-^ 



I t 
-P, #,c,0,g] 

-p,#,0,0,g,0,0,0,0,0,-c] 
-c,0,0,0,0,0,0,0,c] 

-'c,#,0,0,0,0,0,0,c] 

t I 
0, -qf] 



0,0, -#] 



(4) 
(5) 
(6) 
(7) 
(8) 
(9) 
(10) 



For initial counter values m, n G N let 

I{m,n) = agoa^c'"a'^c"a~^. 
The list notation for /(m, n) is 

i t 

[0, go, 0, m • c, 0, 0, 0, n • c]. 



(11) 



12 



Here is some intuition: The group element I{m,n) represents the initial configura- 
tion {qq, m, n) of the 2-counter machine C. Lemma |4]below states that (go, ™, n) 
(g/, 0, 0) is equivalent to the existence of y £ MwithI{m,n)Y = 1, i.e., /(m, n)^^ G 
M. Generators of type (|D-(|8]l simulate the 2-counter machine C. States of C will be 
stored at cursor positions 4fc + 1. The values of the first (resp., second) counter will be 
stored at cursor positions 8fc + 3 (resp., 8fc + 7). Note that /(m, n) puts a single copy of 
the symbol qq ^ S at position 1, m copies of symbol c (which represents counter val- 
ues) at position 3, and n copies of symbol c at position 7. Hence, indeed, I{ni, n) sets 
up the initial configuration (go, ttl, n) for C. Even cursor positions will carry the spe- 
cial symbol Note that generator (|9) is the only generator which changes the cursor 
position from even to odd or vice versa. It will turn out that if /(m, n)Y ^ \(Y ^ M), 
then generator (|9} has to occur exactly once in Y; it terminates the simulation of the 
2-counter machine C. Hence, Y can be written as y = U [qj^ a^^)V with U,V £ M. 
Moreover, it turns out that J7 G A/ is a product of generators (|4)-(|8), which simulate C. 
Thereby, even cursor positions will be marked with a single occurrence of the special 
symbol #. In a second phase, that corresponds to y e M, these special symbols # will 
be removed again and the cursor will be moved left to position 0. This is accomplished 
with generator JTOl i. In fact, our construction enforces that 1/ is a power of (fTOl i. 

During the simulation phase (corresponding to J7 e M), generators of type (|4) im- 
plement zero tests, whereas generators of type (|5) (resp., ^) increment (resp., decre- 
ment) a counter Finally, (|7]i and dHJ copy the counter value to the next cursor position 
that is reserved for the counter (that is copied). During such a copy phase, (|7]) is first 
applied > many times. Finally, ^ is applied exactly once. 

Lemma 4. For all m,n E N the following are equivalent: 

- {qo,m,n) {qf, 0,0) 

— There exists Y £ M such that I{m, n)Y — 1. 

Proof. Assume first that I{m, n)Y = 1 for some Y G M. We have to show that 
{qq, m, n) — {qf, 0, 0); this is the more difficult direction. Let 

where each y.i is one of the generators of M. For < « < fc let 

Yi^yi---yi 

(thus, Yq = 1) and assume that 

I{m,n)Y^ = {fi,mi) e K » Z. 

Hence, fk = is the zero-mapping and nik = 0. Moreover (/o, mo) = I{m, n). 

Claim 1. For all < i < A:, g e Q, and ^ e Z we have fi{2l, q) = 0. 

Proof of Claim 1. Assume that fi{2£, q) ^ for some < i < k, q G Q, and £ € Z. 
Choose < i < k minimal such that there exist q £ Q and I £TL with fi (21, q) ^ 0. 
Since fo{2£, q) — for all g g Q and £ £ "L (the list notation for (/o, too) is (fTTT)). we 
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must have i > 1. Hence, fi^i{2£,q) = for all q e Q and £ e Z. An inspection of 
the generators shows that if TOj_i were odd, we would also have fi{2£, q) = Q for all 
q € Q and £ G Z. Therefore, m^-i must be even. An inspection of the generators of M 
shows that there exist j G Z and p E Q such that 

/,(2j, p) < and h{2j',p') = for all / < j and p' e Q. 
But then, for alH < i' < fc there exist j E Z and p E Q such that 

U (2j, p)<0 and U (2j', p') = for all f < j and p' G Q. 
For i' = kwe obtain a contradiction, since fk = 0. 

Claim 1 implies that for all 1 < i < fc with mi_i even, the generator yi cannot be of 
type dUl, or 

Claim 2. For all < i < fc and £ G Z we have /,(2£, c) = 0. 

Proof of Claim 2. Assume that fi{2£,c) ^ for some < i < fc and ^ G Z. Choose 
< i < k minimal such that there exists ^ G Z with fi{2£, c) ^ 0. Since fo{2£, c) = 
for all G Z, we must have i > 1. Hence, /i_i(2£, c) = for all £ G Z. An inspection 
of the generators shows that if rrii^i were odd, we would also have fi{2£, c) = for 
all ^ G Z. Therefore, rrii-i must be even. The generator yi must be of one of the types 
©, (IHl, (l7]i, or (O. But the types (|5]l and (|6]l are excluded by the remark before Claim 2. 
Therefore, yi must be either (|7]i or (|8). Thus, there exists G Z such that 

/,(2j, c) < and /,(2j', c) - for all / < j. 

Note that for alH < i' < fc with rriii^i even, the generator yi' is not of type Q (again 
by the remark before Claim 2). This implies that for all i < i' < k there exists j G Z 
such that 

(2j, c) < and (2/, c) = for all f < j. 
For i' = kwe obtain a contradiction, since fk = 0. 

Claim 1 and 2 imply that for all 1 < i < fc with TOi_i even, the generator yi is (fTOl l. 

Claim 3. For all < i < fc and £ G Z we have /,(2£ + !,#)= 0. 

Proof of Claim 3. Assume that /,;(2£ + 1, #) ^ for some < i < fc and f G Z. 
Choose < i < A: minimal such that there exists ^ G Z with fi{2£ + 1, #) ^0. Since 
/o(£, #) = for all £ G Z, we must have i > 1. Hence, /»-i(2^ + 1, #) = for all 
£ G Z. There are two possible cases: 

1. mi_i is odd and yi is the generator ( fTOt . 

2. rrii-i is even and is a generator of type (|4}-(|6j or (jS). 

But the second case is not possible by the remark before Claim 3. Hence, mi_i is odd 
and yi is the generator (fTOt . Thus, there exists j G Z with fi{2j + 1, #) < 0. Since 
for every i < i' < k with niii^i even, the generator yi/ can only be of type ( fTOl l 
(again by the remark before Claim 3), it follows that for every i < i' < k we have 
fi' (2j + 1, < 0. For i' — kwe obtain a contradiction, since fk — 0. 
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Claim 4. There is exactly one 1 < i < k such that i/i is the generator (|9}. 
Proof of Claim 4. For g — (/, m) G ) Z and & G {0, 1} we define 

^Q(3,&) = EE/(2fc + fe,<7). 

keZqeQ 

An inspection of all generators of M shows that for every g G Z^?Z and every generator 
z of M we have: 

- If z is not the generator (|9j, then (TQ{gz, b) — (Jqig, b) for both 6 = and 6=1. 

- If z is the generator (|9), then there is 6 G {0, 1} such that aQ{gz, 6) = (Jqig, 6) — 1 
and agigz, 1 - 6) = ctq(5, 1 - 6). 

The claim follows, since (TQ{I{m,n),0) — aQ{I{in,n)Y,0) = aQ{I{m,n)Y,l) — 
and aQ{I{m, n), 1) = 1. 

By Claim 1^, there exists a unique 1 < i < k such that the following three properties 
hold: 

- For every 1 < j < i, yj is a generator of type (|4|l-(|8|l. 

- Ui is the generator (|9). 

- For every i < j < k, yj is the generator (fTO^ . 
Hence, I(m, n)Yi-i must be of the form 

[0, 0, #, 0,#,0, #,..., 0,#,0, #,(?/], 

since only such an element can be reduced to 1 by right-multiplication with gener- 
ator (|9j followed by a positive power of generator (ITOl . We show that this implies 
{qo, m, n) [qj, 0, 0). Note that every generator of type (2)-(l8) (those generators 
that occur in Yi^i) moves the cursor 2d (for some d > 0) to the right along the Z-line. 
This means that for every < j < i — 1, rrij is odd and moreover, for every odd 
m < rrij, the group element fj{m) G Z-^ is zero. 

Claim 5. Let < j < i — I and assume that /(m, n)Yj is of the form 

[0, 0, #,0,#,0, #,..., 0,#,0,#i,0,a-c,0, 0,0,6 -c], (12) 

where p G Q^,a,b G N, and p occurs at position £ = 8fc+l for some fc > (hence. (fT2]l 
represents the configuration (p, a, 6)). Then there exists j' > j and a valid C-transition 
(p, a, 6) -^c {q, b') such that /(m, is of the form 

[0,0, #,0,#,0, #,..., 0,#,0,#,q,0, 6' -c, 0,0, 0,a'-c]. 

t 

Here q occurs at position £ + A. 

Proof of Claim 5. Generator yj+i has to be of the form (|4|i, (|5]l, or (|6]l, because otherwise 
we leave at position i a negative copy of c, which cannot be compensated later Let us 
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first assume that yj+i has the form (|4]i, i.e., {p, = 0,q) £ S. Then I{m, n)Yj+i is of the 
form 

[0, 0, #,0, #, 0, #,..., 0,#,a.c,#,g, 0,6 -0,0,0,0,0], (13) 

t 

where q occurs at position ^ + 4. If a > 0, then the a many c's at position £ + 2 cannot 
be removed in the future. Hence, we must have a — 0. Setting a' = and h' ~h shows 
that (fT3T l has the form required in the conclusion of Claim 5. 

Next, assume that Vj+i has the form (|5]l. Hence (p, +1, q) £ S and /(m, n)Yj^i is 
of the form 

[0, 0,#,0,#,0, #,..., 0,#, (a +\) -0,0,9, 0,6 -0,0,0, 0,0], 
t 

where (a + 1) - c occurs at position £ + 2. So we have to remove a + 1 many copies of c 
from position £ + 2. Hence, the only way to continue is to apply a many times generator 
(|7]i followed by a single application of generator (|8). Hence, /(m, n)lj+a+2 must be 
of the form 

[0,0, #, 0,#,0, #,..., 0,#,0,#,q, 0,6 -c, 0,0,0, (a + l)-c], (14) 

where q occurs at position £ + A. Setting b' = b and a' = a + I shows that ( fT4] l has the 
form required in the conclusion of Claim 5. 

Finally, assume that yj+i has the form (|6]l, hence {p, —l,q) G S and /(m, n)Yj+i 
is of the form 

[0,0, #, 0, #,0, #,..., 0,#,a%,0,g,0, 6 -cO, 0,0, -c], 

t 

where a ■ c occurs at position £ + 2. First, assume that a = 0. Then there is no way to 
move the cursor to the right without leaving a negative copy of a symbol from Q U {c} 
at position £ + 2, and this negative copy cannot be eliminated later. Hence, we must 
have a > 0. Now, the only way to continue is to apply a — 1 many times generator d?) 
followed by a single application of generator (|8). Hence, /(m, must be of the 

form 

[0,0, #, 0,#,0, #,..., 0,#,0,#,g, 0,6 -c, 0,0,0, (a -l)-c], (15) 

where q occurs at position £ + 4. Setting b' ~ b and a' = a — 1 shows that ( fTSl l has the 
form required in the conclusion of Claim 5. 

This concludes the proof of Claim 5. Completely analogously to Claim 5, one can 
show: 

Claim 6. Let < < i — 1 and assume that /(to, n)Yj is of the form 

[0, 0,#,0,#,0, #,..., 0,#,0,#,p,0,a-c,0, 0,0,6- c], (16) 

where p e Qi, a, 6 G N, p occurs at position £ = 8fc + 5 for some fc > (hence, ( fTSl l 
represents the configuration (p, 6, a)). Then there exists j' > j and a valid C-transition 
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(p, b, a) {q, b', a') such that I{m, n)Yj> is of the form 

[0, 0,#,0,#,0, #,..., 0,#,0,#,$,0,&'-c, 0,0, 0,a'-c]. 

t 

Here q occurs at position £ + A. 

Using Claim 5 and 6 we can now easily conclude that {qo,m, n) -^^ (9/1 0, 0) holds. 

The other direction (if {qo,m,n) — >^ (qy,0,0) then there exists Y e M with 
/(to, n)Y — 1) is easier. A computation 

{qa,m,n) -^c ((?i,TOi,ni) -^c ■ ■ ■ -^c (g^-i, to^_i, -^c iqf,^,0) 

can be directly translated into a sequence of Af-generators 2/12/2 ■•• ?/fc such that the 
group element /(to, n)yiy2 ■ ■ - yk has the form 

[0, 0,#,0,#,0, #,..., 0,#,0,#,g/], 

Multiplying this element with generator (|9} followed by a positive power of generator 
JTOl i yields the group identity. □ 

The following result is an immediate consequence of Theorem|4]and LemmaH] 

Theorems. There is a fixed finitely generated submonoid M of the wreath product 
lP I Z with an undecidable membership problem. 

Finally, we can establish the main result of this section. 

Theorem 6. There is a fixed finitely generated submonoid M of the wreath product 
Z I Z with an undecidable membership problem. 

Proof. By Theorem|5]it suffices to reduce the submonoid membership problem of Z^ ;Z 
to the submonoid membership problem of Z ? Z. If to = then Proposition[T]shows 
that Z^ I Z ^ Z™ I toZ is isomorphic to a subgroup of index m in Z ? Z. So if Z i Z 
had decidable submonoid membership for each finitely generated submonoid, then the 
same would be true of Z"^ I Z. □ 

We remark that, together with the undecidability of the rational subset membership 
problem for groups // i (Z x Z) for non-trivial H ||231 . our results imply the following: 
For finitely generated non-trivial abelian groups G and //, the wreath product H I G has 
a decidable rational subset membership problem if and only if (i) G is finit^ or (ii) (G 
has rank 1 and H is finite). Furthermore, for virtually free groups G and H, the rational 
subset membership problem is decidable for H I G if and only if (i) G is trivial or (ii) 
H is finite, or (iii) (G is finite and H is virtually abelian). 

By im, the wreath product Z ; Z is a subgroup of Thompson's group F as well as of 
Baumslag's finitely presented metabelian group (a, s, t | [s,t] — [a*, a] — l,a'' — aa*). 
Hence, we get: 

' If G has size m, then by Proposition[T] /f™ = H"^ll is isomoiphic to a subgroup of index m in 
H I G. Since H"^ is finitely generated abelian, decidability of the rational subset membership 
problem of H I G follows from the fact that decidability is preserved by finite extensions 
fTS.lSJ . 
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Corollary 1. Thompson 's group F as well as Baumslag 's finitely presented metabelian 
group both contain finitely generated submonoids with an undecidable membership 
problem. 

6 Open problems 

As already mentioned in the introduction, we conjecture that the rational subset mem- 
bership problem for a wreath product H I G with H non-trivial and G not virtually free 
is undecidable. Another interesting case, which is not resolved by our results, concerns 
wreath products G I V with V virtually free and G a finitely generated infinite torsion 
group. Finally, all these questions can be also asked for the submonoid membership 
problem. We do not know any example of a group with decidable submonoid mem- 
bership problem but undecidable rational subset membership problem. If such a group 
exists, it must be one-ended (22} . 
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